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BACKGROUND OF THE INVENTION 

The present invention relates generally to haptic feedback provided to humans when 
interfacing with computer systems, and more particularly to haptic sensations and computer 
interface devices that allow the user to provide input to computer systems and allow computer 
systems to provide haptic feedback to the user. 

A user can interact with an environment displayed by a computer to perform functions 
and tasks on the computer, such as playing a game, experiencing a simulation or virtual reality 
environment, using a computer aided design system, operating a graphical user interface (GUI), 
etc. Common human-computer interface devices used for such interaction include a mouse, 
joystick, trackball, steering wheel, stylus, tablet, pressure-sensitive sphere, or the like, that is 
connected to the computer system controlling the displayed environment. Typically, the 
computer updates the environment in response to the user's manipulation of a physical 
manipulandum such as a joystick handle or mouse, and provides visual and audio feedback to the 
user utilizing the display screen and audio speakers. The computer senses the user's 
manipulation of the user manipulandum through sensors provided on the interface device that 
send locative signals to the computer. For example, the computer displays a cursor or other 
graphical object in a graphical environment, where the location of the cursor is responsive to the 
motion of the user manipulandum. 



In some interface devices, force feedback or tactile feedback is also provided to the user, 
more generally known herein as "haptic feedback." These types of interface devices can provide 
physical sensations which are felt by the user manipulating a user manipulandum of the interface 
device. One or more motors or other actuators are coupled to the joystick, mouse, or other device 
5 and are connected to the controlling computer system. In kinesthetic force feedback systems, the 
computer system controls forces on the movement of the joystick or mouse in conjunction and 
coordinated with displayed events and interactions by sending control signals or commands to 
the actuators. For example, the Logitech Wingman Force Feedback Mouse and the Logitech I- 
Feel Mouse allow a user to experience haptic sensations associated with interacting a cursor with 
10 graphical objects in a GUI or web page. 

Low-cost haptic devices tend to provide tactile feedback, in which forces are transmitted 
to a housing or portion thereof and felt by the user, rather than kinesthetic feedback, in which 
forces are output directly in the degrees of freedom of motion of the interface device. For 
:,3 example, many currently-available gamepad controllers include a spinning motor with an 
1^5 eccentric mass, which outputs vibrations to the housing of the controller in coordination with 
events occurring in a game. In some haptic mouse devices, pins, buttons, or the housing of the 
mouse can be vibrated in accordance with interaction of a controlled cursor with other graphical 
^ objects, which the user feels by touching those housing areas. These tactile devices are typically 
O less sophisticated than kinesthetic devices and therefore are much less expensive to produce. 

j*?0 One problem with such inexpensive haptic controllers is their limited ability to convey 

□ certain types of haptic sensations to the user. For example, most pure tactile mice are incapable 
of providing spring and damper sensations in the sensed degrees of freedom of the mouse. These 
types of devices tend to be limited in producing other spatially-based haptic sensations as well. 
This can limit the applicability of tactile mice to interfaces designed for kinesthetic force 
25 feedback devices. For example, many traditional interface elements, such as graphical objects in 
a graphical user interface, tend to be associated with spatially-based force sensations. Icons, for 
example, are often associated with attractive and/or repulsive forces, which apply a force on a 
mouse in its degrees of freedom to assist the user in cursor moving tasks. However, in a tactile 
mouse providing no such forces in the degrees of freedom of the mouse, such haptic sensations 
30 cannot be output. What is needed are haptic sensations to associate with graphical elements in 
interface tasks that are compelling and useful to a user of a tactile mouse or other tactile interface 
device. 
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SUMMARY OF THE INVENTION 



The present invention presents different types of tactile sensations which can be used in a 
variety of applications to convey interesting and useful information to the user of a haptic 
feedback device. 

More particularly, in a first method of the present invention for providing a haptic 
sensation to a user physically contacting a haptic feedback device, where the haptic feedback 
device communicates with a host computer implementing a graphical environment, the method 
includes receiving an indication of an interaction of a user-controlled cursor with a graphical 
object, and enabling the output of a tactile sensation during the interaction, the tactile sensation 
being based on a periodic waveform and having a frequency correlated with a size of the 
graphical object interacted with the cursor. The frequency of the tactile sensation can be 
correlated with a size of a collection of data associated with the graphical object, or correlated 
with a displayed size of the graphical object or other size characteristic. The interaction can be 
moving the cursor over the graphical object or dragging the graphical object. A magnitude or 
duration of the sensation can also be correlated the size of the graphical object. 

In a different aspect of the present invention, a method for providing a haptic sensation to 
a user physically contacting a haptic feedback device includes receiving an indication of a 
position of a user-controlled cursor, causing the cursor to snap to the graphical object when the 
cursor is within a predetermined distance of the graphical object in the graphical environment so 
that the cursor is displayed touching the graphical object and the cursor can be moved along or 
within the graphical object based on user manipulation of the haptic feedback device, and 
enabling the output of a vibration sensation while the cursor is moved along or within the 
graphical object. The graphical object can be a line and the vibration can be enabled to be output 
when the cursor moves along the line. Input from the haptic feedback device based on user 
manipulation that would cause the cursor to move away from the graphical object can be ignored, 
for example, when the input designates cursor locations within a predetermined distance from the 
graphical object. 

In a different aspect of the present invention, a method for providing a tactile sensation to 
a user physically contacting a haptic feedback device includes enabling the output of a pop 
sensation that is a short, high magnitude sensation, and enabling the output of a ringing sensation 
during or immediately after the output of the pop sensation. The ringing sensation is commanded 
using a periodic waveform and has a longer duration and lower magnitude than the pop 
sensation. The ringing sensation can fade in magnitude over time. The haptic sensation can be a 
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rubbery sensation having a ringing sensation using a periodic waveform having a frequency of 
125 to 200 Hz and a duration of about 200 milliseconds, or a metallic sensation having a ringing 
sensation with a frequency of 300 to 500 Hz and a duration of about 200 milliseconds. 

The present invention provides a variety of compelling tactile sensations that can be used 
with a purely tactile feedback interface device. Since tactile devices are typically less complex 
and expensive to produce than kinesthetic force feedback devices, the tactile sensations presented 
herein advantageously extend the applicability and desirability of the less expensive tactile 
feedback devices. 

These and other advantages of the present invention will become apparent to those skilled 
in the art upon a reading of the following specification of the invention and a study of the several 
figures of the drawing. 
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BRIEF DESCRIPTION OF X 




FIGURE 1 is a perspective view of an example of a haptic feedback device and host 
5 computer suitable for use with the present invention; 

FIGURE 2 is a side elevational view of an example implementation of the haptic mouse 
device shown in Fig. 1 ; and 

FIGURE 3 is a diagrammatic view of a displayed line object and a cursor interacting with 
the object according to one aspect of the present invention. 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 



FIGURE 1 is a perspective view of a haptic feedback mouse interface system 10 suitable 
for use with the present invention and capable of providing input to a host computer based on the 
user's manipulation of the mouse and capable of providing haptic feedback to the user of the 
mouse system based on events occurring in a program implemented by the host computer. 
Mouse system 10 includes a mouse 12 and a host computer 14. It should be noted that the term 
"mouse" as used herein, indicates an object generally shaped to be grasped or contacted from 
above and moved within a substantially planar workspace (and additional degrees of freedom if 
available). Typically, a mouse is a smooth or angular shaped compact unit that snugly fits under 
a user's hand, fingers, and/or palm, but can also be implemented as a grip, finger cradle, cylinder, 
sphere, planar object, etc. 

Mouse 12 is an object that is preferably grasped or gripped and manipulated by a user. 
By "grasp," it is meant that users may releasably engage a portion of the object in some fashion, 
such as by hand, with their fingertips, etc. In the described embodiment, mouse 12 is shaped so 
that a user's fingers or hand may comfortably grasp the object and move it in the provided 
degrees of freedom in physical space. For example, a user can move mouse 12 to provide planar 
two-dimensional input to a computer system to correspondingly move a computer generated 
graphical object, such as a cursor or other image, in a graphical environment provided by 
computer 14 or to control a virtual character, vehicle, or other entity in a game or simulation. In 
addition, mouse 12 preferably includes one or more buttons 16a and 16b to allow the user to 
provide additional commands to the computer system. 

Mouse 12 preferably includes an actuator 18 which is operative to produce forces on the 
mouse 12. This operation is described in greater detail below with reference to Fig. 2. 

Mouse 12 rests on a ground surface 22 such as a tabletop or mousepad. A user grasps the 
mouse 12 and moves the mouse in a planar workspace on the surface 22 as indicated by arrows 
24. Mouse 12 may be moved anywhere on the ground surface 22, picked up and placed in a 
different location, etc. A sensor system translates the planar motion of the mouse 12 into 
electrical position signals, which are sent to a host computer 14 over a bus 20 as is well known to 
those skilled in the art. Mouse 12 is preferably a "relative" device, which, as referenced herein, 
is a device that reports a change in position of the device to the host computer rather than an 
absolute position in a fixed reference frame. 

Mouse 12 is coupled to the computer 14 by a bus 20, which communicates signals 
between mouse 12 and computer 14 and may also, in some preferred embodiments, provide 
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power to the mouse 12. Components such as actuator 18 require power that can be supplied from 
an interface such as a USB or Firewire (IEEE 1394) bus. In other embodiments, signals can be 
sent between mouse 12 and computer 14 by wireless transmission/reception. In some 
embodiments, the power for the actuator can be supplemented or solely supplied by a power 
5 storage device provided on the mouse, such as a capacitor or one or more batteries. 

Host computer 14 is a computing device that can take a wide variety of forms. For 
example, computer 14 can be a personal computer or workstation, such as a PC compatible 
computer or Macintosh personal computer, or a Sun or Silicon Graphics workstation. For 
example, the computer 14 can operate under the Windows™ , MacOS, Unix, or MS-DOS 
10 operating system. Alternatively, host computer system 14 can be one of a variety of home video 
game console systems commonly connected to a television set or other display, such as systems 
available from Nintendo, Sega, Sony, or Microsoft. In other embodiments, host computer system 
14 can be a "set top box" which can be used, for example, to provide interactive television 
'i3 functions to users, or a "network-" or "internet-computer" which allows users to interact with a 
\M5 local or global network using standard connections and protocols such as used for the Internet 
: ..Z and World Wide Web. Computer 14 can also be a portable computer or game device, arcade 
\2 machine, vehicle computer, etc. Host computer preferably includes a host microprocessor, 
a random access memory (RAM), read only memory (ROM), input/output (I/O) circuitry, disk 
Q drives, and other components of computers well-known to those skilled in the art. 

Is:- 

5=30 Host computer 14 preferably implements a host application program with which a user is 

£3 interacting via mouse 12 and other peripherals, if appropriate, and which may include force 
! "* feedback functionality. For example, the host application program can be a drawing/CAD 
program, video game, word processor or spreadsheet, Web page or browser that implements 
HTML or VRML instructions, scientific analysis program, virtual reality training program or 

25 application, or other application program that utilizes input of mouse 12 and outputs force 
feedback commands to the mouse 12. Herein, for simplicity, operating systems such as 
Windows™, MS-DOS, MacOS, Linux, Be, etc. are also referred to as "application programs." In 
one preferred embodiment, an application program utilizes a graphical user interface (GUI) to 
present options to a user and receive input from the user. Herein, computer 14 may be referred as 

30 providing a "graphical environment,", which can be a graphical user interface, game, simulation, 
or other visual environment. The computer displays "graphical objects" or "computer objects," 
which are not physical objects, but are logical software unit collections of data and/or procedures 
that may be displayed as images by computer 14 on display screen 26, as is well known to those 
skilled in the art. The host application program checks for input signals derived from the 

35 electronics and sensors of mouse 12, and outputs force values and/or commands to be converted 
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into forces output for mouse 12. Suitable software drivers which interface software with haptic 
feedback devices are available from Immersion Corporation of San Jose, California. 

Display device 26 can be included in host computer 14 and can be a standard display 
screen (LCD, CRT, flat panel, etc.), 3-D goggles, or any other visual output device. Typically, 
the host application provides images to be displayed on display device 26 and/or other feedback, 
such as auditory signals. For example, display screen 26 can display graphical objects from a 
GUI and/or application program. 

Mouse 12 can include a wide variety of architectures. For example, in some 
embodiments, the host computer 14 controls the output of actuators on the mouse 12 directly by 
streaming force values to the mouse 12. In other embodiments, a local microprocessor on the 
mouse 12 can receive high level commands from the host 14 and provide the force values to the 
actuator based on the host commands and local force processes. These implementations, and 
variations thereof, are described in copending application no. 09/456,887, filed 12/7/99, 
incorporated herein by reference. 

In alternative embodiments, the mouse can be a different interface or control device. For 
example, a haptic feedback trackball, a hand-held remote control device, gamepad controller for 
video games or computer games, a knob, cellular phone, personal digital assistant, touchpad, 
stylus, joystick, steering wheel, etc. can be used with the present invention. 

FIGURE 2 is a side cross-sectional view of one embodiment 40 of mouse 12 of Fig. 1. 
Mouse 12 includes one or more actuators 18 for imparting haptic feedback such as tactile 
sensations to the user of the mouse. Suitable embodiments and mechanisms for use as a tactile 
mouse or other tactile pointing device (trackball, gamepad, joystick, stylus, etc.) are well known 
to those of skill in the art. For example, suitable embodiments for a tactile mouse are described 
in Patent No. 6,211,861 and copending application nos. 09/585,741 and 09/759,780, all 
incorporated herein by reference. 

Mouse 12 includes a housing 50, a sensing system 52, and an actuator 18. Housing 50 is 
shaped to fit the user's hand like a standard mouse while the user moves the mouse in the planar 
degrees of freedom and manipulates the buttons 16. Other housing shapes can be provided in 
many different embodiments. Buttons 16 can be selected by the user as a "command gesture" 
when the user wishes to input a command signal to the host computer 14. 

Sensing system 52 detects the position of the mouse in its planar degrees of freedom, e.g. 
along the X and Y axes. In the described embodiment, sensing system 52 includes a standard 
mouse ball 54 for providing directional input to the computer system. Ball 45 is a sphere that 
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extends partially out the bottom surface of the mouse and rolls in a direction corresponding to the 
motion of the mouse on a planar surface 22. For example, when the mouse 12 is moved in a 
direction indicated by arrow 56 (y direction), the ball rotates in place in a direction shown by 
arrow 58. The ball motion can be tracked by a cylindrical roller 60 which is coupled to a sensor 
5 62 for detecting the motion of the mouse. Sensor 62 be any of a variety of types of sensors, such 
as optical encoders, analog potentiometers, etc. A similar roller and sensor 28 can be used for 
the x-direction which is perpendicular to the y-axis. Other types of mechanisms and/or 
electronics for detecting planar motion of the mouse 12 can be used in other embodiments. For 
example, an optical sensor can be used to track mouse motion; a suitable optical mouse 
10 technology is made by Hewlett Packard of Palo Alto, CA and is used in, for example, the 
Intellimouse® Explorer or Intellimouse® with Intellieye™ mouse devices from Microsoft 
Corporation. Use of this sensor is described in greater detail in U.S. Patent No. 6,211,861. 
Alternatively, a portion of an optical sensor can be built into the surface 22 (such as a pad) to 

=.3 detect the position of an emitter or transmitter in mouse 12 and thus detect the position of the 

;;Ji5 mouse 12 on the surface 22. 

t- * An actuator 18 is coupled to the housing 50 to provide haptic feedback to the user. In one 

I s * embodiment of the present invention, an actuator is coupled to an inertial mass that is moved by 
* ! the actuator. The actuator is also coupled to the housing of the mouse such that inertial forces 

□ caused by the motion of the inertial mass with respect to an inertial ground are applied to the 
? t0 housing of the mouse, thereby conveying haptic feedback such as tactile sensations to the user of 

□ the mouse who is contacting the housing. Sensations can alternative or additionally be output 
with reference to an earth ground, e.g. by moving a portion of the housing against the user, as 
described in copending application 09/585,741. One embodiment 40 creates inertial forces that 
are directed substantially in a particular degree of freedom, i.e. along a particular axis. The 

25 inertial forces can be created, for example, using a high bandwidth linear actuator, such as a 
linear moving voice coil actuator and a linear moving-magnet actuator. A flexure or other 
mechanism can be used to convert rotary forces to substantially linear forces in a desired 
direction, such as along the Z-axis, orthogonal to the planar X and Y axes used for mouse motion 
and input. 

30 In the described embodiment, actuator 1 8 can be a linear electromagnetic actuator having 

a stationary portion 66 coupled to the mouse housing 50 (and thus stationary only with respect to 
the portion of the mouse housing to which it is coupled), and a moving portion 67 that moves 
linearly approximately along the Z-axis. In the described embodiment, the stationary portion 66 
includes a magnet and the moving portion 67 includes a wire coil. An inertial mass 64 is coupled 

35 to (or is integral to) the linearly-moving portion of the actuator. The actuator 18 is operative to 
oscillate the inertial mass 64 quickly along the axis C which is approximately parallel to the Z 
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axis. Thus, inertial forces produced by the moving mass 64 are transmitted to the housing 
through the stationary portion 66 of the actuator 18 and felt by the user. These forces are 
substantially directed along the Z axis. Preferably, the mass has an origin position that is 
positioned between the extremes of the range of motion of the mass. In some embodiments, a 
5 spring compliance is built into the actuator or the mass so that the mass returns to the origin 
position when no forces are exerted on the mass by the actuator. 

In other embodiments, moving surface tactile sensations can be conveyed to the user 
instead of inertial sensations. For example, a surface on the mouse, on a housing, a touchpad, or 
other contacted surface can be directly moved in the z-axis against the user's finger or palm. Or, 
10 the contacted surface can oscillated side-to-side in the x-y plane, conveying tactile sensations in 
shear on the user's skin. These types of tactile sensations can also be used with the present 
invention. 

C3 

! 5 Actuator 18 can be a linear voice coil actuator as described in Patent No. 6,243,078, 

rtS which is incorporated herein by reference in its entirety. Alternatively, actuator 18 can be other 
S5 types of actuators/mechanisms, such as a DC motor and flexure. A variety of tactile sensations 

IS *a 

M can be output to the user, many of which are described in Patent No. 6,21 1,861 . A block diagram 
of a haptic feedback system, including local control such a microprocessor or host control, and 
q which can be suitable for the present invention is described in Patent Nos. 6,21 1,861, 6,243,078, 
! =8 and 5,825,308, all incorporated herein by reference in their entirety. 

,"=20 In addition, it is preferred that the mouse rest on a flexible or semi-flexible surface, such 

M as a standard mouse pad. This type of flexible surface increases the transmissibility of the 
inertial forces from the actuator to the housing and allows the inertial sensations to be amplified, 
rather than countered by normal forces applied by a hard surface such as a table surface. An 
amount of flex can also be provided between the actuator-coupled portion of the mouse housing 
25 and the base portion 68 that is in contact with the surface 22 to improve the transmissibility of 
the tactile sensations. 



Tactile Sensations 

Tactile sensations applied to the user operating a tactile feedback interface device can 
30 take a variety of forms. Tactile sensations of the present invention which have been found to be 
particularly suitable for interface tasks, such as tasks within a graphical user interface or 
application program, are described below. 
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Any of the tactile sensations described below can be controlled by a driver program 
running on the host computer, by an application program or API on the host computer, or by the 
local microprocessor on the haptic feedback device, or by a combination of two or more of these. 
For example, a driver program on the host (at a lower level than the application program) can 
check when a cursor interacts with an object and cause the necessary commands to be output to 
implement a tactile sensation to the haptic device. Or, an application program on the host can 
command the haptic sensations directly. Alternatively, a local microprocessor or other controller 
can check cursor interactions and command sensations, as long as the local microprocessor has 
an up-to-date picture of the layout of the graphical environment, e.g. the host can send data to the 
microprocessor conveying this information. 



Conveying a Sense of Object Size 

In kinesthetic force feedback devices of the prior art, forces can be output in the degrees 
of freedom of the mouse. Thus, for example, a displayed icon can be associated with an 
attractive force, where forces are output in the direction of the icon to draw the mouse toward the 
icon. 

In other situations, a sense of the size of a graphical object (or an associated collection of 
data) can be conveyed when the user uses the mouse to move the graphical object or move the 
cursor over the object. For example, in Patent No. 6,219,032, incorporated herein by reference in 
its entirety, a user can move a cursor on some types of graphical objects and "drag" them, e.g. 
move the objects with the cursor while holding down a mouse button. In kinesthetic feedback 
devices, to make the objects feel as if they have weight, a force can be output in conjunction with 
the drag operation, such as an inertia force or resistance force in the opposite direction to the drag 
direction. To convey a size of the object, the magnitude of the force can be in proportion to the 
size of the object. Thus, a smaller graphical object will provide a smaller resistance force, while 
a larger graphical object provides a larger resistance force. The force magnitude can also or 
alternatively be indicative of a collection of data, program, or other quantity or feature associated 
with the graphical object. For example, the force for a folder can be proportional to the number 
of files stored within that folder, and/or proportional to the size (e.g. in bytes) of a file within that 
folder. In other kinesthetic force feedback embodiments, an attractive force can bias the device 
or manipulandum to move the cursor toward an object, and the attractive force magnitude can be 
scaled in proportion with the size of object or size of an associated characteristic. 

Purely tactile mice, however, such as the embodiment described with reference to Fig. 2, 
cannot output forces in the degrees of freedom of the mouse and thus cannot output drag forces, 
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inertia forces, attractive forces, or the like. However, haptic sensations can be output to convey 
to the user a size of graphical object which is being manipulated or moved over with the cursor. 

Tactile sensations can be conveyed by commanding a periodic waveform to the actuator. 
For example, a sine wave that moves positive and negative can cause the actuator to move an 
5 inertial mass in positive and negative directions about an origin position, thereby conveying a 
vibration sensation. A square wave, triangle wave, sawtoothed-shaped wave, or other types of 
periodic waveforms can also be commanded. Many different types of vibration sensations and 
other sensations can be designed and output using such software tools and programs as 
Immersion Studio™ and Immersion Desktop™, available from Immersion Corp. 

10 The duration, or number of periods in the commanded wave, determines how long the 

vibration sensation is output. If a small number of high frequency periods of a waveform are 
Q commanded, it typically feels like a single "pulse" or jolt of force to the user, since the oscillatory 
motion occurs too quickly for the user to notice. 

*■<. .j 
I y 

Such pulses can be associated with graphical objects in a graphical user interface, 
;;15 application program, or other displayed graphical environment. For example, when the user 
i fl interacts the cursor with a graphical object, a short duration periodic waveform is preferably sent 
" to the mouse, causing a short vibration or "pulse" to be experienced by the user. This pulse can 
5fl be output when the cursor enters a graphical object or exits the object, or when the cursor moves 
i»~ over the middle of the object (e.g., for smaller graphical objects). Graphical objects associated 

with such forces can include menu items, icons, buttons, tool bars, slider thumbs, radio buttons, 
\*& check boxes, and hyperlinks or other objects in a web page. Web page force sensations are 

described in U.S. Patent No. 5,956,484, incorporated herein by reference in its entirety. 

According to the present invention, a size of the graphical object can also be conveyed 
using tactile sensations. The frequency and duration of the waveform can be adjusted to convey 

25 this size. For example, when the user moves the cursor over a small menu item, a tactile 
sensation is output by commanding a high frequency vibration to the device which preferably has 
a short enough duration to convey a small number of cycles (or periods) of the vibration, such as 
two to five. For example, a waveform having a frequency of 125 Hz and a duration of 14 
milliseconds can convey this sensation. To the user, this feels like a small jolt or pulse on the 

30 mouse which indicates the graphical object has been engaged or interacted with the cursor. As 
another example, the sensation of an even smaller object, such as a radio button, can be conveyed 
by commanding a periodic waveform having a very high frequency and a very short duration, 
such as 300 Hz for 5 milliseconds. These vibrations are a high enough frequency as to feel like a 
single jolt or pulse to the user. A low magnitude for the vibration can also be specified to 

35 enhance the "small" perception to the user, but magnitude typically is not as important as 
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frequency, since the user is typically able to adjust the magnitude (gain) of all forces output on 
the mouse device. Distinguishing a small object by frequency is independent of any such gain 
control and thus more reliable. 

When the user moves the cursor over a larger graphical object, such as a large toolbar 
5 icon or a desktop icon, a waveform having a lower frequency is preferably commanded. To 
convey the preferable 2-5 cycles of vibration, the waveform will typically also have a longer 
duration than the higher, frequency waveforms for smaller graphical objects. A waveform 
having, for example, a frequency of 83 Hz and a duration of 25 milliseconds can be used. This 
provides the sensation to the user of moving the cursor over or into a large bump or other 
10 obstruction, i.e. the lower frequency and longer duration pulse conveys the feeling of a larger 
object. 

q The pulses or vibrations for different sizes of objects can be varied to a fine degree. For 

^ example, the frequency and duration of the vibrations can be adjusted in 10 Hz and 10 
rjj millisecond increments according to the size of the associated graphical object, or a more gradual 
Ms adjustment can be used. When experiencing the different feels of all of the graphical objects, the 
\Z user correlates the different force sensations with differently-sized objects so that the user 
W intuitively knows the sizes of graphical objects simply by the tactile sensations associated with 
^ those objects. 

•a t? 

u In other embodiments, the frequency and duration of a commanded waveform can convey 

];i0 the size of associated data, application program, or other characteristic. For example, a displayed 
\nl graphical icon can be associated with a file having a size in bytes. A high frequency waveform, 

as described above, can be commanded to output a vibration for smaller file sizes associated with 

icons, while lower frequency waveforms can be used to command pulses for larger file sizes. 

Similarly, the size or importance of an associated application program can be similarly conveyed 
25 via frequency and duration. The different pulses can also be made to convey the importance of a 

graphical object with regard to the preferences of a particular user. 



Tactile Drawing Application Sensations 

Drawing, graphics, and computer-aided design (CAD) application programs allow a user 
30 to input and edit line drawings, bitmaps, and other pictorial works displayed on the computer 
display device. In many types of line drawing applications, a user can move a cursor over a line, 
shape, or other drawing object and select the object with the cursor. The user can then 
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manipulate or edit the object in several ways, such as move or delete the object, change the shape 
or facing of the object, resize the object, change the color of the object, etc. 

Existing kinesthetic force feedback mouse implementations output forces in the planar 
degrees of freedom of the mouse and thus can provide forces that directly assist the user in 
selecting a desired drawing object and tracing over that object. For example, a line object can be 
associated with attractive forces so that when the cursor is moved within a predetermined 
distance of the line object, the attractive force is activated, causing the mouse to be biased in a 
direction toward the line object. When the cursor is positioned on or very near the line object, 
the attractive forces act to resist movements of the cursor away from the line object. This 
provides a "snap to" sensation that allows the user to select the line object more easily. In 
addition, the user can directly feel the shape of the line by moving the cursor along the line 
object. If the user moves the mouse away from the line, the forces bias the mouse back toward 
the line so that the user can more easily trace the line with the cursor. 

In purely tactile haptic devices, no such attractive or snap-to forces can be output in the 
degrees of freedom of the mouse or other manipulandum. However, according to the present 
invention, forces can be output to inform and assist the user in selecting and tracing drawing 
objects or the like. In one preferred embodiment, when the cursor is moved within a 
predetermined distance of an object such as a line, the cursor is visually caused to snap to the 
line, i.e., the host computer displays the cursor at the line, which may cause the cursor to move 
on the screen without any user motion of the mouse. An example is shown in FIGURE 3, which 
shows a line 100 displayed on a display screen. If the cursor moves within a specified distance of 
the line 100, indicated by dashed line 102 (preferably not displayed, but can be displayed in 
alternate embodiments), the cursor 104 is displayed at the line 100 as shown. The distance can 
vary in different embodiments, and even be specifiable by the user as one or more preferences in 
an application program. 

In one embodiment, once the cursor is positioned on the line, the user may move the 
mouse to move the cursor along the line. If mouse motions cause the cursor to move along the 
line, then the display of the cursor is not modified. If, however, the mouse is moved in a 
direction that would cause the cursor to move away from the line, then the mouse input in that 
30 direction is ignored. This causes the cursor to remain on the line regardless of mouse input. For 
example, if the cursor is on a line that is horizontal, then mouse motion in a diagonal direction 
(horizontal plus vertical components) would cause the cursor to move along the horizontal line, 
but not in any vertical direction. Preferably, only mouse motion within a predetermined radius of 
the snap-to object is filtered in such a way; this radius can be the same as the entering distance 
35 (indicated by line 102), or can be made a different distance. In alternate embodiments, the cursor 
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can be allowed to move within the area defined by line 102 before the mouse inputs are filtered 
or ignored (e.g. the non-line mouse motions can be ignored out to a distance from the line 100 
greater than line 102). 

According to the present invention, while the cursor is snapped to the line, a vibration 
5 sensation is preferably output while the cursor is engaged with the line (e.g., while the cursor is 
positioned to select the line). The vibration is preferably a high frequency vibration that indicates 
to the user that the cursor is engaged with or positioned over or near the line. The vibration can 
also have a lower magnitude. To the user, the line then feels as if it were a "live wire," e.g., as if 
he or she were touching a wire having electricity flowing through it, or as if touching a cord 
1 0 having sound vibrating through it. This indicates to the user that the line may be selected, but is 
not so strong a force sensation as to cause a disturbance to the user or impede the cursor motion. 
Furthermore, it is a very compelling sensation to move the cursor along the line and feel the 
h i vibration. When the cursor moves off the line, the user is immediately aware that selection is no 
O longer possible. 

2 5 hi other embodiments, a different vibration can be commanded. For example, one or 

ujl more of the vibration characteristics can be made proportional to the current velocity of the 

(pes ..... 

! = ! - mouse in its workspace. In one embodiment, low mouse velocities cause a vibration having a 

==2 lower magnitude and/or frequency, while higher mouse velocities cause a vibration having a 

£3 higher magnitude and/or frequency. 

|;20 It should be noted that other types of graphical object which are traced by the cursor can 

**J! St? 

M be correlated with vibrations as described above. For example, the outlines of polygons or other 
shapes can be implemented similarly to the line object described above. Furthermore, 
background lines or shapes, such as grid lines, may be associated with such haptic sensations in 
some applications. 

25 

Rubbery and Metallic Interactions 

In many force feedback interfaces to graphical environments, it is desirable to create 
unique and individual haptic sensations that can indicate needed information to the user 
concerning an interaction of the controlled cursor with another displayed object, or concerning 
30 some other event. 

One type of haptic sensation that can be both compelling and informative to the user is a 
"rubbery" or "metallic" sensation. To the user, these types of sensations feel like tapping a finger 
or a held object on a rubber-like ringing material, or a metallic ringing material, where ah amount 
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of ringing or vibration occurs after a tap. Displayed objects such as icons, menus, toolbars, 
hyperlinks, or checkboxes can be provided with such a feel. For example, a user may prefer a 
rubbery sensation when moving a cursor between menu items or when moving the cursor onto an 
icon. 

5 In addition, certain types of objects can be associated with this type of haptic sensation to 

distinguish those types of objects from other types of objects. The rubbery and metallic feels can 
also be used to distinguish some interface functions from other functions, e.g. distinguishing the 
selection of a button from the selection of a drop down menu in an interface. Events, such as the 
reception of email or a calendar reminder, can also be distinguished with the rubbery or metallic 
10 feels. In some embodiments, a user can designate his or her own types of objects or instances of 
objects with the sensations or variations thereof. Some applications such as games or 
simulations can use the rubbery or metallic feels to provide particular effects when the user is 
interacting with objects in a graphical environment, such as when a controlled character 
3 encounters a surface or other object. Alternatively, game events can be distinguished by the 
| rubbery or metallic sensations, such as when a player is being attacked by another player. One 

-2 tyP e °f event or interaction can be provided with a rubbery feel, while another type can be 
i»* provided with a metallic feel or a more standard vibration or pulse. 

|U An effective way of providing such a rubbery or metallic feel to a displayed object for a 

'•s Si? 

C3 tactile interface device is to combine a "pop" sensation with a reverberating "ring" sensation. 

The pop sensation is a short jolt or "pulse" of force which preferably has a high amplitude 
O relative to the ringing sensation. The pop sensation can be implemented in many different ways. 
For example, a basic implementation is to move an inertial mass with a high amplitude, e.g. a 
high amplitude amount of current is sent to a motor for a short period of time. In other 
implementations, the pop can be provided by commanding one or a small number of periods of a 

25 high magnitude vibration. If a small number of high frequency periods of a waveform are 
commanded, it typically feels like a single "pulse" or jolt of force to the user, since the oscillatory 
motion occurs too quickly for the user to notice. The pop sensation is preferably output first 
when the cursor first interacts with the target, to signify to the user the engagement of the cursor 
with the target, or is output at the beginning of the event notification associated with the rubbery 

30 or metallic sensation. 

A ring sensation follows the pop and provides the rubbery or metallic ringing 
characteristic of the sensation, e.g. while the cursor is interacting with an associated target or 
during an associated event. The ring is a long vibration that is of a lower magnitude than the pop 
to provide a more subtle effect to the user. In preferred embodiments, the ring sensation fades in 
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magnitude over time. To achieve a metallic feel, a higher frequency vibration is preferably used 
for the ring sensation. To achieve a rubbery feel, a lower frequency vibration is preferably used. 

Some examples of the described sensations are provided below. For a rubbery sensation 
associated with a graphical object (or associated with some other event), the following haptic 
5 sensations can be characterized: 

—pop: a short, high magnitude pulse or vibration 

—ring: a periodic waveform having a duration of 200 milliseconds. The frequency 
of the waveform is preferably in the range of 125 Hz to 200 Hz. The waveform can be, 
for example, a square wave, sine wave, triangle wave, or other type of wave. The initial 
10 magnitude is preferably about 40% of the magnitude of the initial pop sensation. The 

iS3% magnitude preferably fades over the 200 millisecond period, the ending magnitude being 

q less than 20% of the magnitude of the initial pop. 

i/ 3 ! To provide a metallic haptic sensation associated with a graphical object (or another 

„5 event), the following characteristics can be defined: 

is ss 

: <IS 

5 —pop: a short, high magnitude pulse or vibration 

□ 

f?i —ring: a periodic waveform having a duration of about 200 milliseconds. The 

M frequency of the waveform is preferably in the range of 300 Hz to 500 Hz. Square, sine, 
triangle, or other types of waves can be used. The initial magnitude is preferably about 

U 10-20% of the magnitude of the initial pop sensation. The magnitude preferably fades 

20 over the 200 millisecond period, the ending magnitude being nearly zero. 



In some embodiments, the ring sensation can start immediately after the pop sensation 
has been output. In other embodiments, the ring sensation can start at the same time as the pop 
sensation, or soon after but before the pop sensation is finished. Since the pop sensation has a 
25 very short duration, a user cannot distinguish the difference in the ring sensation start times. The 
most important characteristic is that a high magnitude, fast jolt is followed by the softer, fading 
reverberation sensation. 

The sensations described above can be designed and output using such software tools and 
programs as Immersion Studio™ and Immersion Desktop™, available from Immersion Corp. 

30 While this invention has been described in terms of several preferred embodiments, it is 

contemplated that alterations, permutations and equivalents thereof will become apparent to 
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those skilled in the art upon a reading of the specification and study of the drawings. For 
example, many different embodiments of haptic feedback devices can be used to output the 
tactile sensations described herein, including joysticks, steering wheels, gamepads, and remote 
controls. Furthermore, certain terminology has been used for the purposes of descriptive clarity, 
and not to limit the present invention. It is therefore intended that the following appended claims 
include alterations, permutations, and equivalents as fall within the true spirit and scope of the 
present invention. 
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